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[57] ABSTRACT 

Refinement of motion block motion vectors to achieve a 
dense motion field for temporal video signal processing 
makes use of candidate motion vectors from a parent block 
and neighboring child block groups to determine the motion 
vector for a child block within one of the child block groups. 
The vicinity of each candidate motion vector may be 
searched to obtain a trial motion vector having a minimum 
displaced frame difference measure, with the trial motion 
vector with the minimum displaced frame difference mea- 
sure from among all the trial motion vectors being the 
motion vector for the child block. Alternatively each can- 
didate motion vector may be applied to the child block, and 
the vicinity of the candidate motion vector having the 
minimum displaced frame difference being searched to 
obtain the motion vector for the child block. If the minimum 
displaced frame difference is not less than a given threshold, 
the child block is segmented and the process repeated until 
the child block size is a single pixel or the minimum 
displaced frame difference is less than the threshold. 

3 Claims, 2 Drawing Sheets 
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REFINEMENT OF BLOCK MOTION 
VECTORS TO ACHIEVE A DENSE MOTION 
FIELD 

BACKGROUND OF THE INVENTION 

The present invention relates to video signal processing 
including video compression, temporal interpolation and the 
like, and more particularly to a method of refining block 
motion vectors to achieve a dense motion field. 

Video and many medical images are received as 
sequences of two-dimensional image frames or fields. To 
transmit such images as digital signals some form of com- 
pression is required. Three basic types of redundancy are 
exploited in a video compression process: temporal 
redundancy, spatial redundancy and amplitude redundancy. 
Interframe coding techniques make use of the redundancy 
between successive frames (temporal redundancy). In these 
techniques the information defining elements of a picture, 
Le., pixels, are estimated by interpolation or prediction using 
information from related locations in preceding and/or suc- 
ceeding versions of the picture, as exemplified in U.S. Pat 
No. 4383J272 issued May 10, 1983 to Netravali et ai entitled 
<( Vldeo Signal Interpolation Using Motion Estimation." A 
typical compression encoder is shown in FIG. 1 where a 
preprocessed video signal is input to a motion estimator. The 
motion estimator delays the video signal, to compensate for 
the processing delays for the motion vector generation 
process, before providing the video signal to an encoder loop 
where compression is performed. The compression is per- 
formed using a motion vector generated by the motion 
estimator, which is multiplexed with the compressed video 
signal at the output of the encoder loop for transmission. 

The interpolation between frames in the compression 
encoder is performed by first estimating the motion 
trajectory, Le., motion vector or displacement vector, of each 
pixeL If an estimate of such displacement is available, then 
more efficient prediction may be performed in the encoder 
by relating to elements in a previous frame mat are appro- 
priately spatially displaced. These displacement vectors are 
used to project each pixel along its trajectory, resulting in the 
motion compensated prediction or interpolation. Once the 
motion vectors are determined, then the differences between 
consecutive motion compensated frames that exceed a pre- 
determined threshold are determined by the encoder loop as 
the compressed video signal. 

Most motion estimation in mterframe coding assumes (i) 
objects move in translation, i.e., zoom and rotation are not 
considered, (ii) illumination is spatially and temporally 
iinifonru and (iii) occlusion of one object by another and 
uncovered background are not considered. In practice 
motion vectors are estimated for blocks of pixels so that the 
displacements are piecewise constant Block matching is 
used to estimate the motion vector associated with each 
block of pixels in a current coding frame or field, assuming 
that the object displacement is constant within a small 
two-dimensional block of pixels. In these methods the 
motion vector for each block in the current frame or field is 
estimated by searching through a larger search window in a 
previous frame/field and/or succeeding frame/field for a best 
match using correlation or matching techniques. The motion 
estimator compares a block of pixels in the current frame 
with a block in the previous or future frame by computing 
a distortion function, such as shown in FIG. 2. Each block 
in the current frame is compared to displaced blocks at 
different locations in the previous or future frame within a 
search window, and the displacement vector that gives the 
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minimum value of the distortion function is selected as 
being the best representation of the motion for that block. A 
motion vector refinement generator, as shown in FIG. 2. 
further processes the selected motion vector to provide a 

5 final motion vector output 

For motion compensated image sequence interpolation in 
the temporal domain, a dense motion field, i.e., a motion 
field resolved on a per pixel level, is required in order to 
have no blocking artifacts. For example if motion vectors of 

10 an NxM pixel block are used to carry out the motion 
compensated video sequence interpolation, blocking arti- 
facts are most likely visible around objects which move 
independendy with respect to each other and/or to the 
background. When two objects, portions of which occur 

15 within a single search block, are moving relative to each 
other, one of the objects constitutes a majority of the pixels 
in the block. The resulting block motion vector most likely 
tracks the majority object motion, so that part of the block 
which doesn't belong to the majority object but rather to the 

20 minority object has the wrong motion vector associated 
with it In other words blocking artifacts occur due to the fact 
that block segmentation of the motion field does not corre- 
spond to a perfect per object segmentation in the video 
sequence. 

25 Block based motion compensated compression systems 
are well established as described above, such as the MFEG-1 
and MPEG-2 compression standards. For motion compen- 
sating video sequence interpolation, the simple search 
method based solely on minimizing the displaced frame 

30 difference (DFD) measure often fails to produce good visual 
results. Hierarchical block matching techniques have been 
used to produce a homogeneous motion vector field, but the 
hierarchical approach lacks resolution in complex scenes. A 
statistical Gibbs/Markov model has been used to maximize 

35 the probability that the derived motion vector field repre- 
sents the "true" physical motion of the scene, but this 
approach is often stochastic in nature and involves iterated 
computations. Quad-tree type decomposition/segnnentation 
schemes also are well known for two-dimensional (2D) 

40 image data, which may be coupled with motion estimation 
to derive the motion vector field, but the sub-divided block 
again goes through the same motion estimation procedure as 
the parent block so an important neighborhood constraint is 
not used. 

45 

In order to resolve the motion field boundary along an 
arbitrary object edge, the motion field needs to be segmented 
along the boundary in a dense fashion, with per pixel motion 
field vectors being deemed dense enough for interpolation 
50 purposes. 

SUMMARY OF THE INVENTION 

Accordingly the present invention provides a method of 
refining block motion vectors to achieve a dense motion 

55 field. A parent block is subdivided, or segmented, into four 
child block groups. For any child block within a group there 
are three ntighboring child block groups. The motion vec- 
tors for the parent block and each neighboring child block 
group are determined and selected as a set of candidate 

60 motion vectors for the child block. The motion vector search 
is restricted to a vicinity around each of the candidate motion 
vectors. Hie resulting motion vector with the minimum 
displaced frame difference (DFD) measure is the motion 
vector for the child block. Alternatively each candidate 

65 motion vector is applied to the child block and the one 
selected for search in the vicinity is the one with the 
minimum DFD measure. In either case if the DFD measure 
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is within a certain threshold, then that motion vector 
becomes the fine motion vector for the child block. Other- 
wise the child block groin? of the child block becomes the 
parent block and the process is repeated until the desired fine 
resolution as determined by the threshold is reached. Thus 
the motion vector for an object is determined by the shape 
of the object, and not by an arbitrary block configuration. 

The objects, advantages, and other novel features of die 
present invention are apparent from the following detailed 
description when read in conjunction with the appended 
claims and attached drawing. 

BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 is a block diagram view of a prior art video 
compression system which may include refining block 
motion vectors to achieve a dense motion field according to 
the present invention. 

FIG. 2 is a block diagram view of a prior art motion 
estimator which may include refining block motion vectors 
to achieve a dense motion field according to the present 
invention. 

FIG. 3 is an illustrative view of segmentation of a motion 
vector field using block matching according to the present 
invention. 

FIG. 4 is a functional block diagram view for refining 
block motion vectors according to the present invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

A top to bottom quad tree-like sub-division process is 
used to refine the motion vector field from coarse to fine 
resolution. As shown in FIG. 3 each NxM pixel block 10, 
hereto referred to as a parent block, is first divided into four 
equal blocks 12. hereto referred to as the child groups. A new 
motion vector is found for each child block 14 within each 
group. 

Each child block group 12 goes through the same sub- 
division process repeatedly untQ the child block reaches the 
desired fine resolution* ie., down to pixel resolution if 
necessary. In order to maintain a smooth motion vector field, 
the new motion vector for each child block needs to be 
constrained by the local motion vector field. To satisfy the 
local motion vector field constraint: 

(1) Motion vector search for the child block is restricted 
to the vicinity around a set of candidate motion vectors. A 
candidate motion vector is defined as one of the motion 
vectors of (a) the parent block, (b) the child block groups 
neighboring the child block, and (c) optionally the weighted 
average of the parent and neighbor blocks. FIG. 3 shows the 
neighboring child block groups R1-R4 for each position of 
the child block X within the child block group in a quad 
tree-type sub- division. The vicinity of the motion vector (dx. 
dy) is defined as in the range of 

(f-W*)SJXS(mti*)) 

and 

where m and n are small* non-negative integers, dx and dy 
define the candidate motion vector, and dX and dY are the 
motion vector search range for the child block. As indicated 
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in FIG. 4 the new motion vector for the child block is found 
by selecting the motion vector with minimum displaced 
frame difference (DFD) measure for the child block, such as 
the mean absolute difference (MAD) or means square dif- 
5 ference (MSD). 

(2) To save some computation time, instead of searching 
around the vicinity of all the candidate motion vectors, as 
defined above in (1). each candidate motion vector is applied 
to the child block and the one selected is the one with the 

10 minimum DFD measure. Then to improve the estimate still 
further a search is conducted around the vicinity of the 
selected motion vector and the final motion vector is derived 
for the child block. 

(3) The sub-dividing process is stopped when the DFD 
13 measure is small enough, i.e., less than a certain threshold, 

otherwise the sub-dividing process continues until the 
desired fine resolution is reached. 

Thus the present invention provides for refinement of 
block motion vectors to achieve a dense motion field by 
30 using only the neighboring motion vectors for refinement, 
and by stopping the refinement process when the motion 
vector is deemed good enough to prevent further breaking 
up of the motion field. 
What is claimed is: 
25 1. A method of refining block motion vectors for a video 
image sequence to achieve a dense motion field comprising 
the steps of: 

subdividing a parent block of pixels from the video image 
sequence into a plurality of child block groups; 
30 for each child block within a child block group, searching 
for a motion vector for the child block according to a 
defined measure as a function of candidate motion 
vectors associated with the parent block and each 
neighboring child block group; and 
35 repeating the subdividing and searching steps with each 
child block group as the parent block until the defined 
measure for the motion vector is less than a specified 
threshold or the child block represents a single pixel 
from the video image sequence, the resulting motion 
40 vector being the refined block motion vector for the 
child block. 

2. The method as recited in claim 1 wherein the searching 
step comprises the steps of: 

45 selecting one of the candidate motion vectors; 

searching in a vicinity around the selected candidate 
motion vector for a trial motion vector having a mini- 
mum displaced frame difference measure as the defined 
measure; 

so repeating the selecting and search step for each of the 
candidate motion vectors; and 
alerting as the motion vector the trial motion vector 
having the minimum displaced frame difference. 

3. The method as recited in claim 1 wherein the seaiching 
55 step comprises the step of. 

applying each of the candidate motion vectors for the 
parent and neighboring parent blocks to the child block; 
selecting the candidate motion vector from the applying 
step with the minimum displaced frame difference 
60 measure as the defined measure; and 

searching in the vicinity of the selected candidate motion 
vector to obtain the motion vector. 

***** 
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